The invention relates generally to computer operating systems, and deals more particularly with a computer operating system which recovers resources after failure of a commit process involving the resources.
"This patent application is related to U.S. patent applications:
U.S. patent application Ser. No. 07/525,430, entitled "LOG NAME EXCHANGE FOR RECOVERY OF PROTECTED RESOURCES" filed May 16, 1990 by M. K. Ainsworth et al.;
U.S. patent application Ser. No. 07/526,471, entitled "OPTIMIZATION OF COMMIT PROCEDURES" filed May 16, 1990 by A. Coleman et al.;
U.S. patent application Ser. No. 07/525,427, entitled "COORDINATED SNYC POINT MANAGEMENT OF PROTECTED RESOURCES" filed May 16, 1990 by M. K. Ainsworth et al.;
U.S. patent application Ser. No. 07/525,429, entitled "ASYNCHRONOUS RESYNCHRONIZATION OF A COMMIT PROCEDURE" filed May 16, 1990 by K. H. Britton et al.;
U.S. patent application Ser. No. 07/525,939, entitled "REGISTRATION OF RESOURCES FOR COMMIT PROCEDURES" filed May 16, 1990 by A. Coleman; and
U.S. patent application Ser. No. 07/526,472, entitled "COORDINATED HANDLING OF ERROR CODES AND INFORMATION DESCRIBING ERRORS IN A COMMIT PROCEDURE" filed May 16, 1990 by E. A. Pruul et al."
U.S. patent application Ser. No. 07/525,426, entitled "LOCAL AND GLOBAL COMMIT SCOPES TAILORED TO WORK UNITS" filed May 16, 1990 by B. A. Maslak et al.
The operating system of the present invention can be used in a network of computer systems. Each such computer system can comprise a central, host computer and a multiplicity of virtual machines or other types of execution environments. The host computer for the virtual machines includes a system control program to schedule access by each virtual machine to a data processor of the host, and help to manage the resources of the host, including a large memory, such that each virtual machine appears to be a separate computer. Each virtual machine can also converse with the other virtual machines to send messages or files via the host. Each virtual machine has its own conversation monitor system (CMS) portion of the system control program to interact with (i.e., receive instructions from and provide prompts for) the user of the virtual machine. There may be resources such as shared file system (SFS) and shared structured query language (SQL) relational databases which are accessible by any user virtual machine and the host.
Each such system is considered to be one real machine. It is common to interconnect two or more such real machines in a network, and transfer data via conversations between virtual machines of different real machines. Such a transfer is made via communication facilities such as AVS Gateway and VTAM facilities ("AVS Gateway" and "VTAM" are trademarks of IBM Corp. of Armonk, N.Y.).
An application can change a database or file resource by first making a work request defining the changes. In response, provisional changes according to the work request are made in shadow files while the original database or file is unchanged. At this time, the shadow files are not valid. Then, the application can request that the changes be committed to validate the shadow file changes, and thereby, substitute the shadow file changes for the original file. A one-phase commit procedure can be utilized. The one-phase commit procedure consists of a command to commit the change of the resource as contained in the shadow file. When resources such as SFS or SQL resources are changed, the commits to the resources can be completed in separate one-phase commit procedures. In the vast majority of cases, all resources will be committed in the separate procedures without error or interruption. However, if a problem arises during any one-phase commit procedure some of the separate commits may have completed while others have not, causing inconsistencies. The cost of rebuilding non-critical resources after the problem may be tolerable in view of the efficiency of the one-phase commit procedure.
However, a two-phase commit procedure is required to protect critical resources and critical conversations. For example, assume a first person's checking account is represented in a first database and a second person's savings account is represented in a second database. If the first person writes a check to the second person and the second person deposits the check in his/her savings account, the two-phase commit procedure ensures that if the first person's checking account is debited then the second person's savings account is credited or else neither account is changed. The checking and savings accounts are considered protected, critical resources because it is very important that data transfers involving the checking and savings accounts be handled reliably. An application program can initiate the two-phase commit procedure with a single command, which procedure consists of the following steps, or phases:
(1) During a prepare phase, each participant (debit and credit) resource is polled by the sync point manager to determine if the resource is ready to commit all changes. Each resource promises to complete the resource update if all resources successfully complete the prepare phase i.e. are ready to be updated. PA0 (2) During a commit phase, the sync point manager directs all resources to finalize the updates or back them out if any resource could not complete the prepare phase successfully.
An IBM System Network Architecture SNA LU6.2 sync point architecture (reference SC31-6808, Chapter 5.3 "Presentation Services--Sync Point Verbs", published by IBM Corp.) was previously known to coordinate commits between two or more protected resources. This architecture previously addressed sync point facilities consisting of a sync point manager which performed both sync point and associated recovery processing running in a single application environment. Several applications could run simultaneously in this environment. The LU6.2 architecture supports a sync point manager (SPM) which is responsible for resource coordination, sync point logging and recovery. The prior art CICS/VS (trademark of IBM Corp. of Armonk, N.Y.) environment supports such an architecture.
According to the SNA LU6.2 architecture prior art, in phase one and in phase two, commit procedures are executed and the sync point manager logs the phase in the sync point log. Also, the sync point manager logs an identification number of a logical unit of work which is currently being processed. Such logging assists the sync point manager in resource recovery or resynchronization in the event that a problem arises during the two-phase commit procedure. If such a problem arises subsequent to entering the two-phase commit procedure, the log is read and resource recovery processing takes place to bring resources involved to a consistent state. The problems include failure of a communication path or failure in a resource manager.
The aforesaid LU6.2 sync point architecture is defined as one application environment, and every LU6.2 sync point environment runs applications for that environment. Data is typically owned by that environment and not shared outside of the environment, unless it is specifically extracted from the environment. The LU6.2 sync point architecture defines a sync point manager (SPM) model for resource coordination, recovery and sync point manager logging in a single environment. Different environments require separate sync point managers, which include separate sync point and recovery operations, and separate logs, even on the same physical processor. Similarly, in another prior art system control program, CICS/VS, sold by IBM Corp., all processes and most resources are owned by the environment rather than by the system.
In another prior art system control program, VM/SP Release 5, sold by IBM Corp. for supporting multiple (virtual machine) execution environments, two application programs could run in the same (virtual machine) execution environment. However, if the called application program committed file updates, this would cause the calling application program's file updates to be committed even if the files of the calling application program were not yet in a consistent state. There was no feature in this prior art system control program to separate the work of the calling application program from the work of the called application program. In addition, commits were limited to files and, through separate procedures, databases.
In a subsequent prior art system control program, VM/SP Release 6, also sold by the IBM Corp. for a virtual machine environment, two application programs running on the same virtual machine (execution environment) could define different work units for their files. As a result, the files accessed by one application could be committed independently of the files accessed by the other application, and the work of one application could be done independently of the work of the other application. Also, in this subsequent prior art system control program, one application (for example, a server) could have multiple work units concurrently. Nevertheless, this subsequent prior art system control program was limited in that, although multiple resources could have the same work unit, the resource updates had to be committed independently. Furthermore, work unit was confined to an execution environment (virtual machine).
According to prior art there is one recovery facility associated with each sync point manager, in the same execution environment as the sync point manager and the protected application; thus potentially there are multiple recovery facilities for a system that supports multiple execution environments. Likewise, each sync point manager requires a separate sync point log for supporting recovery from sync point failures. Thus, for each execution environment in systems that support applications participating in coordinated recovery (sync point), there is a sync point manager that includes a recovery facility and a sync point recovery log. Also, if the execution environment that runs the application experiences a failure, it is difficult to support recovery.
In prior art, commits were coordinated with two types of resources; data resources, each represented by a resource manager, and application partners, each represented by a conversation with that application partner. The application partner, in turn, coordinated commits with other data resources etc. During a recovery operation, sync point state information is exchanged between the syncpoint managers representing the application partners (assuming the sync point managers are operable).
In prior art, a failure of an application or its partner, or the connecting conversation and communications facilities, requires a predefined, unilateral decision for resolving sync points that are in the second stage of the two-phase commit. This is where a commit decision had been given to the sync point participants, but confirmation of the commit had not been received from all participants (an "in-doubt" state). In such cases, a unilateral recovery decision could result in damage to participating resources where the resources were changed inconsistently (one participating resource was committed while another was backed out). For example, assume a sync point was in an in-doubt state when a failure occurred, such that a first participating resource manager had received the commit request from the sync point manager and processed the commit of its resource, while a second resource manager, participating in the same sync point, had not processed the commit. If the second resource manager, in the interest of clearing the in-doubt state so that normal resource processing could continue, decides to unilaterally back out the prepared sync point, thereby undoing any changes to the second resource, there results an inconsistency between the first and second resource relative to the intent of the using application(s). Such actions on the part of the second resource manager are not uncommon because resources are frequently locked during the sync point to prevent other users of the resource from accessing the resource until it consistently changed and committed. In the CICS system if one application partner or the conversation to it fails, then the sync point manager automatically attempts continuous retries for a limited time period, and if unsuccessful, then makes a heuristic decision to commit or back out.
The prior art also permitted participating resource managers that fail during a sync point, after regaining operability, to notify their sync point manager so that the sync point manager can complete recovery of any outstanding sync points for that resources. However, this notification by the resource manager requires manual activation of a resource-specific transaction or command, processed by the sync point manager. Furthermore, in the prior art, this only applied to resource managers, not to other sync point managers (protected conversations) that are participants in the sync point. Such notification is only possible where the originating sync point manager is not also involved in the failure.
A general object of the present invention is to provide an automatic recovery facility to complete failed sync points in systems that support multiple execution environments.
Another general object of the present invention is to provide an efficient recovery facility and log architecture.
It is a further object of the present invention to avoid manual administrator intervention wherever possible, but when necessary, to provide for administrator input to resolve failed sync point correctly in lieu of automatic resolution through programmed protocols.
It is a further object of the present invention to extend the automatic recovery facility to support an open-ended set of resources and resource managers, and to support the recovery of participating resource managers without requiring operator or administrator involvement, except for extended outages.
It is a further object of the present invention to expedite recovery from a failed sync point.